The present invention relates generally to multi-node computer systems, and more specifically to a mechanism for decoding a destination node address of a memory request in a multi-node computer system.
Multi-node computer networks may include central processor unit (CPU) nodes, memory nodes, input/output (I/O) nodes, and hybrid nodes (with any combination of memory, I/O, and CPU). These nodes are connected via a system interconnect, which is responsible for address decoding, i.e., for determining to which node a request should be routed.
In multi-node computer systems, memory has high latency compared to present day CPU speeds. This means that the time for a memory node to respond to a read or write request is large. Another frequent bottleneck is the maximal throughput, i.e., the amount of data a memory system can provide per unit time. Memory interleaving is a well-known technique that allows a multi-node computer system to increase throughput by splitting the memory system across a group of nodes at an interleave size. For example, in a system with four memory nodes with an interleave size of x, a base address B can be mapped to Node 0. Address B+x is mapped to Node 1, B+2x to Node 2, B+3x to Node 3, and B+4x to Node 0. This allows the system to avoid any hot memory spots as well as to increase the system performance.
As multi-node computer systems are becoming larger, it becomes important to be able to address many nodes. Existing methods require base and size (limit) declaration for each node in the system. Thus, if there are n nodes in the system, they require n base registers and n size registers. As the number of nodes increases, the memory registers holding the {base, size} pairs increase linearly, thereby requiring very large amount of chip real estate.
Another disadvantage of existing methods is that in order to determine quickly the destination node address of the request, existing solutions require multiple magnitude comparators. There is typically one magnitude comparator for each node. As the number of nodes is added, more {base, size} pairs must be added and more magnitude comparators are needed. The cost of implementation of these magnitude comparators is usually very high. Thus, existing decode schemes are not scalable enough to support many address nodes.
Yet another disadvantage of conventional implementations having multiple nodes and using interleaving, is that conventional systems use a fixed interleave size and a limited number of combination of nodes for each interleave group.
To summarize, existing decode schemes are not scalable enough to support many address nodes. In addition, existing address decoding schemes are not flexible enough to allow different sizes for interleaving and to allow a variety of interleave sets.
What is needed, therefore, is an improved mechanism for address space decoding in a multi-node computer system.
The present invention includes a system for address space decoding in a multi-node computer system. In accordance with an embodiment of the present invention, a multi-node computer system includes a plurality of I/O nodes, CPU nodes, memory nodes, and hybrid nodes connected by an interconnect (as shown in FIG. 1). In one embodiment of the present invention, a request issued by a CPU node or an I/O node includes an address comprising a base field that stores a base address of a destination node; an index field that stores a logical address of a destination node; and a granularity field that stores a size of an addressable space of a memory node.
The system further includes an address decoder adapted to extract a base address of a destination node, using a width of the base field. The address decoder is also configured to extract a logical address of a destination node, using a width of the index field and the granularity field. The address decoder further comprises a base register for storing a number of bits indicating the width of the base field; an index register for storing a number of bits indicating the width of the index register; and a granularity register for storing a number of bits indicating a width of the granularity field. The width of the granularity field is used to determine where the index field starts in the address. The address decoder further comprises a base offset register for storing a programmed base offset indicating where a memory node is mapped in a system address space; a logical comparator for performing a comparison between the base address and the base offset to determine whether the request for data is made to a memory node; and a mapping table for mapping the extracted logical address of a destination node to a physical node address where the request is routed.
The present invention also includes a method for address space decoding in the multi-node computer system. Initially, a messaging driver causes a CPU node or an I/O node to issue a request to a memory node. The request includes an address. The address decoder extracts a base field of the address using the width of the base field. A logical comparison is performed between the base address and the programmed base offset. If the two match, it indicates that the request is a coherent memory request, i.e., it is made to a memory node. If the request is a coherent memory request, the address decoder extracts a logical node address of the destination node using the width of the index field and the granularity field. A physical destination node address is determined based on the logical node address by indexing into the mapping table.
The present invention advantageously performs a logical comparison instead of performing arithmetic comparisons to decode a destination of the request. This obviates the need of having multiple magnitude comparators. Furthermore, as the number of nodes in a multi-node computer system increases, the memory registers do not increase linearly. As a result, the implementation cost does not scale linearly, but remains small.